Computational Simulations and the Scientific Method

AS scientific simulation software becomes more complicated, the scientific-software implementer’s need for component tests from new model developers becomes more crucial. The community’s ability to follow the basic premise of the Scientific Method requires independently repeatable experiments, and model innovators are in the best position to create these test fixtures. Scientific software developers also need to quickly judge the value of the new model relative to other models, i.e., the new model’s cost-to-benefit ratio in terms of gains provided by the new model and risks such as implementation time and software quality. This letter asks two questions. The first is whether other scientific software developers would find published component tests useful, and the second is whether model innovators think publishing test fixtures is a feasible approach. In Refs. 1 and 2, we argue that as computational models become more complex, software unit-testing practices become essential for advancing simulation capabilities. Those papers called for model and algorithm innovators to publish succinct test fixtures, i.e., sample input and output, so that subsequent implementers can independently verify they have correctly translated the new innovation to source code, i.e., so the Scientific Method’s notion of independently-verifiable experiments can be used. This letter provides an alternative presentation of those ideas in light of copious feedback. As growth in computational power facilitates higher-fidelity computational simulation techniques, the number and variety of building-block components also increases. While this increased complexity is forcing a change from the cottage industry of one person/one code to team software development to address increasing software system size,3 the community is not yet routinely publishing independently verifiable tests for new models or algorithms to address the code-verification complexity. The survey results of Table 1 show that only 22% of new models published are accompanied by tests suitable for independently verifying the new model. To sustain our growing numerical simulation capability, we need to become competent software developers;4 and one measure of software development competence is sound software testing practices.5,6 For example, before inserting a new component into a system, software developers will perform a set of component-level tests. Based on feedback from previous conference presentations,1,2 many agree with the need for component-level testing in the computational simulation community but there is disagreement about how to implement it. While each development group could derive component-level tests for each model they choose to implement, this duplication is unnecessary and would not likely catch the special cases that the original innovator would likely know intimately. Besides, the Hatton studies of scientific codes underscores the difficulty in achieving consistent implementations: 1 fault per 170 lines.7,8 This letter calls for institutionalizing component-level testing in the computational simulation community and offers one possible route toward implementation. The letter begins by exploring the current practice, recalls basic tenets of the Scientific Method, proposes a course of action, gives a couple brief examples, and finishes with some concluding remarks.